New Type of Degenerate Quantum Spin Phase 
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Correlated electrons often crystalize to the Mott insulator usually with some magnetic orders, 
whereas the "quantum spin liquid" has been a long-sought issue. We report numerical evidences 
that a nonmagnetic insulating (NMI) phase gets stabilized near the Mott transition with remark- 
able properties: The 2D Mott insulators on geometrically frustrated lattices contain a phase with 
gapless spin excitations and degeneracy of the ground state in the whole Brillouin zone of the total 
momentum. It has an interpretation for an unexplored type of a quantum liquid. The present con- 
cept is useful in analyzing a variety of experimental results in frustrated magnets including organic 
BEDT-TTF compounds. 
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Among various insulating states, those caused by elec- 
tronic Coulomb correlation effects, called the Mott insu- 
lator, show many remarkable phenomena such as high-Tc 
superconductivity and colossal magnetoresistance near 
it However, it has also been an issue of long debate 

whether the Mott insulator has its own identity distin- 
guished and adiabatically separated from other insulators 
like the band insulator. This is because the Mott insu- 
lator in most cases shows symmetry breakings such as 
antiferromagnetic order or dimerization, where the resul- 
tant folding of the Brillouin zone makes the band full 
and such insulators difficult to distinguish from the band 
insulators because of the adiabatic continuity. 

Except in one dimension, the possibility of the inher- 
ent Mott insulator without conventional orders has been 
a long-sought challenge. The Mott insulator on the trian- 
gular lattice represented by the Heisenberg spin systems 
was proposed as a candidate, where the spins quantum 
mechanically melt against spin solidifications Q]- Al- 
though, the triangular Heisenberg system itself has been 
controversial and argued theoretically to show an antifer- 
romagnetic (AF) order 0|, intensive studies on geomet- 
rical frustration effects have been stimulated. 

In this letter, we show a numerical evidence for the 
existence of a new type of inherent Mott insulator near 
the Mott transition; singlet ground state with unusual 
degeneracy in the total momentum accompanied by gap- 
less and dispersionless spin excitations. 

Recently extensive experimental studies on frustrated 
quantum magnets such as those on triangular, Kagome, 
spinel and pyrochlore lattices have been performed ]^ |^ . 
They tend to show suppressions of magnetic orderings 
with large residual entropy with a gapless liquid feature 
for quasi 2D systems or "spin glass-like" behavior in 3D 
even for disorder- free compounds. These gapless and de- 
generate behaviors wait for a consistent theoretical un- 
derstanding. Our present results offer a useful underlying 
concept for the understanding of the puzzling feature. 

To get an insight into this issue, our study focuses on 
the Hubbard model on two-dimensional frustrated lat- 
tices. The Hamiltonian by the standard notation reads 



[A] 



[B] 



FIG. 1: Lattice structure of geometrically frustrated lattices 
[A] on a square lattice and [B] on an anisotropic triangular 
lattice. The nearest- and next-nearest-neighbor transfers are 
denoted by t and t' , respectively 
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on a iV-site square lattice with a nearest neighbor {t) 
and diagonal next-nearest neighbor [t') transfer integrals 
in the configurations [A] and [B] illustrated in Fig. 
The energy unit is taken by t. The insulating phase is 
stabilized when the on-site Coulomb repulsion U is large 
enough, while the metallic phase appears at small U when 
t' is nonzero as we will show. 

The model requires accurate and unbiased theoretical 
calculations because of large fluctuation effects expected 
from the low dimensionality of space and the geometri- 
cal frustration effects due to nonzero t' . Recently, the 
path-integral renormalization group (PIRG) method 
opened a way of numerically studying the models with 
the frustration effects more thoroughly without the nega- 
tive sign problem and without relying on the Monte Carlo 
sampling. The efficiency of the method was established 
through a number of applications 0, ITol [Tl| . By using 
the PIRG method, the existence of a nonmagnetic insu- 
lator (NMI) near the metal-insulator transition bound- 
aries was reportred H, |^ on the two-dimensional frus- 
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FIG. 2: (color) Phase diagrams of the Hubbard models with 
the lattice structure illustraterd in Fig. 1 [A] and [B] in the 
parameter space of U scaled by t, and the frustration param- 
eter t' /t. AFI, PM, and NMl represent the antiferromagnetic 
insulating, paramagnetic metallic and nonmagnetic insulating 
phases, respectively. The dashed curve indicated by "HF" in 
the upper panel is the Hartree Fock result 



trated Hubbard model on lattices [AJ and [B] . In Fig. 12 
the phase diagrams are illustrated The phase dia- 

grams show quantum melting of spin orders at higher U 
than the Mott transition. This new aspect is ascribed to 
enhanced charge fluctuations and increasing double occu- 
pation near the Mott transition, which cannot be studied 
in the Heisenberg models. However, the nature of the re- 
sultant NMI in 2D remained totally unclear. We reveal a 
couple of remarkable features of this phase in this letter. 

We study the excitation spectra calculated by the 
PIRG method. More specifically, by following the PIRG 
method 0, the wavefunction with the form \^) = 
'^j^i Cj\(j)j) is numerically optimized so that the en- 
ergy estimate (^'l^'} becomes the lowest among 
choices of arbitrary nonorthogonal Slater basis functions 
{(jjj) and the coefficients Cj. The optimization of 
and Cj are achieved by repeated operations of exp[— riJ] 
to The convergence to the true lowest energy state 
is obtained after the systematic increase of L. In the 
present work, we have improved the original PIRG al- 
gorithm to obtain the lowest energy state with spe- 
cific quantum numbers. Such quantum number projec- 
tion can be performed through a rotation of the state 



1^") with an angle (f> in the spin space using the rota- 
tion operator TZ{(f>) and by a spatial translation with 
a shift of r by the translation operator C{r) in the 
process of PIRG. A state with specific quantum num- 
bers S and k is obtained from the weighted integration 
as mS,k)) = /d0W^^(^,0)i?(0)SrW^i(k,r)£(r)|M/). 
Here the weights Wb. and Wl are chosen to specify the 
quantum numbers S and k. For example, Il^L(k, r) = 
exp[ik • r]. Since the z component of the total spin, 5^, 
is fixed in |^), the integral over the Euler angle is re- 
duced to that over a single variable 0. This quantum 
number projection procedure also improves the accuracy 
of the energy estimate substantially. The accuracy of this 
method was carefully examined and confirmed in many 
examples. For example, all the ground states and excita- 
tion spectra studied on 4 x 4 lattice show roughly 4 digit 
accuracy in comparison with the available exact results. 

Before reporting proprties in the NMI phase, we first 
discuss our PIRG result in the AFI phase. Here, the 
finite si ze g ap for a, ix£ system in the chiral perturbation 
theory [13 in the form 
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are fitted with the calculated results in Fig. O The finite- 
size gap nicely follows the form (0) in the AFI phase. For 
example, at J7 = 4, t = 1 and t' — 0, the fitting in Fig. |3 
shows the spin wave velocity c 0.67 and the spin stiff- 
ness p I /7.1c, which are equivalent to the estimate of 
the Heisenberg model at the exchange coupling J — 0.42 
in the spin wave theory. The fitted values of c and J 
well reproduce the previous estimates (ex. J 0.4) ob- 
tained from the susceptibility and the staggered magneti- 
zations 14] . The excitations in the AFI phase well satisfy 
the tower structure of the low-energy excitation spectra 
based on the nonlinear sigma model description. 

Now in the NMI phase, typical system size depen- 
dences of the spin excitation gap AE between the singlet 
ground state and the lowest triplet state are shown in 
Fig. 21 All of the data points in Fig. ^ indicate that the 
triplet excitations become gapless in the thermodynamic 
limit. The gap appears to be scaled asymptotically with 
the inverse system size A^~^, namely AE C/^- The 
gapless feature shares actually some similarity to the be- 
havior in the AFI phase. However detailed comparison 
clarifies a crucial difference as we will show later. The 
fitting of the data in the NMI phase to the form (2) gives 
unphysical values such as c > 1.5. We note that the uni- 
form magnetic susceptibility is given by 2/3C. Therefore, 
the present data imply that the uniform susceptibility 
becomes a nonzero constant. 

Except in ID systems, the present result is the first 
numerical evidence by unbiased calculations for the exis- 
tence of gapless excitations without apparent long-ranged 
order in the Mott insulator. Although a tiny order can- 
not be excluded if it is beyond our numerical accuracy, in 
the present NMI phase, the absence of various symmetry 
breakings including the AF order has already been shown 
in the model [B] p. In the model [A], the size scaling 
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FIG. 3: (color) Size scalings of the energy gaps for total spin 
5=1,2,3 and 4 in the AFI phase (?7 = 4, t = 1, i' = 0). The 
solid curves are fitting by the form (2) and the dashed curves 
illustrate curves obtained from the 5* = fitting multiplied 
with the factor 45(5' + l)/3. 
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FIG. 4: (color) Size scalings of the 5 = 1 excitation gaps for 
several choices of parameters in the NMI phase. The triangles 
show the case of the model [B] while others are for the model 
[A]. The dashed curves are fittings to Eq.(2). The circles, 
squares and triangles are results in the NMI phase. 



suggests the absence of AF order at any wavevector as 
well 0- As well as dimer and plaquette singlet orders, 
s- and d-density waves are also numerically shown to be 
unlikely for four types of correlations probed by (q) = 

|(>^a(q)4(q))|, Ja{<i) = ^Ek,^4,^ck+q.<T/c(k), with 

/i(k) = cos(fc2;) + cos(A:y), /2(k) = cos(fca:) - cos{ky), 
/3(k) = 2 cos(fc3;) cos(fcy), and /4(k) = 2 sin{kx) sm{ky) . 
Fig. O in the example of a = 2 clarifies the correlation of 
the d-density wave (namely, the staggered flux) [THj l. 

The dispersions of the 5 = 1 excitations show dra- 
matic difference between the AFI and NMI phases. Here 



FIG. 5: (color) Size scalings of the staggered fiux correlations 
for the model [B] at t' /t = 1.0 
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FIG. 6: (color) Size scalings of the dispersion widths for 5 = 
and 5 = 1 excitations in the NMI phase for the model [A]. 



the dispersions are given from the lowest energy states 
with specified momenta, £^(k) calculated from the spin- 
momentum resolved PIRG. In the AFI phase, the dis- 
persion is essentially described by the spi n-wave spec- 
trum of the form similar to Ai?(fc) = 47-^1 — 7^ with 
Ik = \ {cos{kx) + cos{ky)) for the spin wave theory of the 
Heisenberg model, but modified because of finite U. The 
calculated dispersion width (~ 1.5) is comparable to the 
estimate of the spin- wave dispersion at J = 0.4. 

In marked contrast, the S = 1 dispersion in the NMI 
phase has strong and monotonic system size dependence 
as in Fig. El For systems larger than 8x8 lattice, the dis- 
persion surprisingly becomes vanishingly small. The size 
dependence shows very quick collapse of the dispersion 
with increasing system size and may not be fitted by a 
power of the inverse system size as in the single-particle 
Stoner excitations in metals. Such a flat dispersion is ob- 
served solely in the NMI phase irrespective of the models 
(namely commonly seen in the models [A] and [B]). 

The presence of such degenerate excitations well ac- 
counts for the quantum melting of simple translational 
symmetry breakings including the AF long-ranged or- 
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der, because any type of long-ranged order in the two- 
dimensional systems is destroyed when the excitation be- 
comes flatter than AE{k) = k^. This is because of the in- 
frared divergence in the form J k'^~^dk-^^^jj^ with d — 2 
when one calculates the fluctuations around the order. 

In addition to S" = 1 excitations, the total singlet state 
{S — 0) at any total momentum k also shows degenerate 
structure in the ground state for larger system size as in 
Fig. El The dispersion vanishes in the NMI phase in the 
model [A] as well as in [B]. 

The present excitation spectra show the following 
double-hierarchy structure: The ground states are de- 
generate within the total spin S — sector among differ- 
ent total momenta; the dispersion quickly collapses with 
increasing system size. Another degeneracy, the gapless 
spin excitation among different total spins, emerges much 
slowly with increasing system size. 

Here we discuss a possible interpretation of the prop- 
erties. The vanishing gap may allow an interpretation 
that the dynamical singlet bonds filling the whole lattice 
contained in the ground state wavefunction has a nonzero 
weight of distribution at vanishingly small singlet bind- 
ing energy, thus the distribution of the singlets over long 
distance with weights of presumable power law decay as 
in a variational long-ranged RVB wavefunction |l6j |. 

The collapse of dispersion shows that a ground state 
is degenerate with other ground states obtained by spa- 
tial translations and they have vanishing off-diagonal 
Hamiltonian-matrix elements each other. This degener- 
acy implies the orthogonality catastrophe under spatial 
translations. Although the translational symmetry is re- 
tained in the original Hamiltonian, the present orthogo- 
nality supports an unconventional symmetry breaking in 
the whole Brillouin zone of the total momenta. The ex- 
citations cannot be coherent propagative modes but are 
localized because of the lack of dispersion. Coherent (or 
Stoner-type) spinon excitations with a spinon Fermi sur- 
face do not seem to explain the very quick collapse of the 
dispersion with the increasing system size as well. From 
the long-ranged RVB picture, one can argue that an un- 
bound singlet does not coherently propagate because of 
scattering by other dynamical singlets. 

This nonmagnetic insulator appears to be stabilized 
simply due to the Umklapp scattering The contin- 
uum of degenerate excitations within the singlet sector, 
which is similar to the present results, but in the pres- 
ence of the spin gap was proposed in the Kagome and 



pyrochlore lattices based on small cluster studies [19j- 
The possible symmetry breaking from degenerate singlet 
states was also examined on a pyrochlore lattice |23, |21| , 
while the spin excitations were again argued to be gapful 
in contrast to the present results. In our results, the de- 
generacy becomes clear only at larger system sizes than 
those in these studies. 

We briefly discuss experimental implications of the 
present new quantum phase. Recent results by Shimizu 
et al. ^3 K-(ET)2Cu2(CN)3 appear to show an ex- 
perimental realization of the quantum phase we have dis- 
cussed in the present work. In fact this compound can 
be modelled by a single band Hubbard model on nearly 
right triangular lattice near the Mott transition. The 
NMR relaxation rate shows the nonmagnetic and gapless 
nature retained even at low temperatures (~ 0.2K) and 
suggests the present quantum phase category. Another 
organic compound also shows a similar behavior psf 

Systems with Kagome-like structure, ^He on 
graphite H and volborthite Cu3V207(OH)2-2H20 [H 
show nonmagnetic and gapless behaviors. On the other 
hand, glass-like transitions are seen in 3D systems, 
typically in pyrochlore compounds as i?2Mo207 with 
R =Er, Ho, Y, Dy, and Tb [26j and in fee structure, 
Sr2CaRe06 ^23]. It is remarkable that the glass behavior 
appears to occur without introducing the disorder. 
Although the lattice structure, dimensionality and local 
moments have a diversity, many frustrated magnets 
show gapless and incoherent (glassy) behavior. The 
present result on gapless and strongly degenerate struc- 
ture emerging without quenched randomness offers a 
consistent concept with this universal trends. It would 
be an extremely interesting and open theoretical issue 
whether the present unusual translational symmetry 
breaking leads to such a glass phase at T = in 2D. 

In summary, we have theoretically clarifled the exis- 
tence of a new degenerate quantum spin phase in the 
Mott insulator under geometrical frustration effects. The 
phase has gapless spin excitations from the degenerate 
ground states, and furthermore the dispersionless modes 
are found in all the spin sector. This may cause an un- 
usual translational symmetry breaking. Recent experi- 
mental flndings, though diverse, tend to show a relevance 
of this phase in disorder-free and frustrated systems. 

A part of the computation was done at the supercom- 
puter center in ISSP, University of Tokyo. 
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